Stagewise Weak Gradient Pursuits Part II: Theoretical Properties

In a recent paper [2] we introduced the greedy Gradient Pursuit framework. This is a family of algorithms designed to find sparse solutions to underdetermined inverse problems. One particularly powerful member of this family is the (approximate) conjugate gradient pursuit algorithm, which was shown to be applicable to very large data sets and which was found to perform nearly as well as the traditional Orthogonal Matching Pursuit algorithm. In Part I of this paper [3], we have further extended the Gradient Pursuit framework and introduced a greedier stagewise weak selection strategy that selects several elements per iteration. Combining the conjugate gradient update of [2] with this selection strategy led to a very fast algorithm, applicable to large scale problems, which was shown in [3] to perform well in a wide range of applications. In this paper we study theoretical properties of the approach. In particular, we propose a novel fast recursion to calculate the conjugate gradient update direction and present a proof that shows that this update is guaranteed to be better than a simple gradient update. The other contribution of this paper is to derive theoretic guarantees that give conditions under which the stagewise weak selection strategy can be used to exactly recover sparse vectors from a few measurements, that is, guarantees under which the algorithm is guaranteed to find the sparsest solution.